K. Frick^ and M. Grasmair^ 

-"^ Institute for Mathematical Stochastics 
University of Gottingen 

GoldschmidtstraBe 7, 37077 Gottingen, Germany 
^Computational Science Center 
University of Vienna 

Nordbergstrai3e 15, 1090 Vienna, Austria 

E-mail: frick@niath.uni-goettingen.de, markus . grasmairOunivie . ac. at 

Regularization of Linear Ill-posed Problems by the 
Augmented Lagrangian Method and Variational 
Inequalities 

Abstract. We study the application of the Augmented Lagrangian Method to 
the solution of linear ill-posed problems. Previously, linear convergence rates 
with respect to the Bregman distance have been derived under the classical 
assumption of a standard source condition. Using the method of variational 
inequalities, we extend these results in this paper to convergence rates of lower 
order, both for the case of an a priori parameter choice and an a posteriori choice 
based on Morozov's discrepancy principle. In addition, our approach allows 
the derivation of convergence rates with respect to distance measures different 
from the Bregman distance. As a particular application, we consider sparsity 
promoting regularization, where we derive a range of convergence rates with 
respect to the norm under the assumption of restricted injectivity in conjunction 
with generalized source conditions of Holder type. 



AMS classification scheme numbers: 65J20, 47A52; 

1. Introduction 

We aim for the solution of the problem 

inf J{u) s.t. Ku — g, (1) 

where K: X ^ H is a linear and bounded mapping between a Banach space X and a 
Hilbert space H and where J: X — > M is convex and lower semi-continuous. We are 
particularly interested in the case when the right hand side in the linear constraint is 
not at hand but only an approximation such that 

||5-5l<^ (2) 

for some (5 > 0. A possible method for computing a stable approximation of solutions 
of (HI) is the augmented Lagrangian method (ALM), an iterative method that, for a 
given initial value pg £ H and for fc = 1, 2, . . ., computes 
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Uj, G argmm 

uex 



^\\Ku~g'i' + J{u) - {pi_,,Ku - g') 



(3 a) 

Pl=pl^^+Tkig' -Kul). (36) 
Here, {Tfe}j,gf5j denotes a pre-defined sequence of positive parameters such that 

n 

tn ■— Tfc -> oo as n — >■ oo. 
fe=i 

The ALM was originally introduced in (Hestenes 1969, Powell 1969) (under the name 
method of multipliers) as a solution method for problems of type ([T]) with exact right 
hand side g. Since then, the ALM was developed further in various directions; see e.g. 
(Fortin & Glowinski 1983, Ito & Kunisch 2008) and the references therein. 

In the context of inverse problems, the ALM was first considered for the special 
case when X is a Hilbert space and J is a quadratic functional, i.e., J{u) ~ ^ ll^wlj^ 
for a densely defined and closed linear operator L: D{L) C X —i' H, where H is some 
further Hilbert space (here we set J{u) = +oo if m ^ D{L)). For this special case, it 
is readily seen that the ALM can be rewritten into 



argmm 

uex 



Tk \\Ku - g^\\^ + \\L{u - w^_i)||^ 



(4) 



The analysis of iteration @ dates back to the papers (Krasnosel'skii 1960, Krjanev 
1973). The case when i = Id is referred to as the iterated Tikhonov method and 
has been studied in (Lardy 1975, Brill & Schock 1987, Hanke & Groetsch 1998, Engl 
et al. 1996). The regularization scheme that results for if = Id is termed iterated 
Tikhonov-Morozov method and amounts to stably evaluate the (possibly unbounded) 
operator L at g given only an approximation g^ that satisfies ^ . For detailed analysis 
see e.g. (Groetsch & Scherzer 2000, Groetsch 2007). 

A generalization of the iteration in Q for total-variation based image 
reconstruction has been established in (Osher et al. 2005) under the name Bregman 
iteration and convergence properties were studied in (Burger et al. 2007). In (Frick 
& Scherzer 2010) it was pointed out that the Bregman iteration and the iterated 
Tikhonov(-Morozov) method are special instances of the ALM as it is stated in Pa|) . 
and an improved convergence analysis was developed. In (Frick et al. 2011), Morozov's 
discrepancy principle (Morozov 1967) was studied for the ALM. The application of the 
ALM for the regularization of nonlinear operators has been considered in (Bachmayr 
& Burger 2009, Jung et al. 2011). 

Up to now, convergence rates for the ALM (in the context of inverse problems) 
have only been derived under the assumption that the solutions of ([U satisfy the 
standard source condition (Burger & Osher 2004) 

K*p'^ e dJ{v)) for some € H. (5) 

Here K*: H — X* denotes the adjoint operator of K and dJ{u'^) is the subdifferential 
of J at . This typically results in a convergence rate of 5 with respect to the Bregman 
distance (for a definition of the subdifferential and the Bregman distance, see Section 
[2]). In this paper we will extend these results to convergence rates of lower order by 
replacing ([5]) by variational inequalities. The analysis will apply for both a priori and 
a posteriori parameter selection rules, where the latter will be realized by Morozov's 
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discrepancy principle. In addition, our approach allows the derivation of convergence 
rates with respect to distance measures different from the Bregman distance. 

The paper is organized as follows: In Section [5] we state basic assumptions and 
review tools from convex analysis that are essential for our analysis. In Section |3] 
we establish variational inequalities and prove that these are sufficent for lower order 
convergence rates for the ALM with suitable a priori stopping rules. In Section [4] we 
reprove the same convergence rates when Morozov's discrepancy principle is employed 
as an a posteriori stopping rule. In Section [5] we finally consider some examples that 
clarify the connection of the variational inequalities in Section [3] and more classic 
notions of source conditions, such as the standard source condition ([5]) or Holder- 
type conditions. Moreover, we show for the particular scenario of sparsity promoting 
regularization how our approach can be used to derive convergence rates with respect 
to the norm. 



2. Assumptions and Mathematical Prerequisites 

In this section we fix some basic assumptions as well as review basic notions and 
facts from convex analysis. We start by delimiting minimal functional analytic 
requirements. 

Assumption 2.1. (i) X is a separable Banach space with topological dual X*. We 

denote the duality pairing of X and X* by {£,,x)-^, ^ = ^{x). 
(a) The operator K.X ^ H is linear and continuous. 

(Hi) The functional J: X W :— R U {+00} is convex, lower semicontinuous and 

proper with nonempty domain D{ J) = {u Cz X : J{u) < 00}. 
(iv) For each g (z H and c > the set 

Aig,c) = [ueX : \\Ku-gf + Jiu)<c^ 

is sequentially weakly pre-compact in X . 

For our analysis we will make extensive use of tools from convex analysis (here, 
we refer to (Ekeland & Temam 1976) as a standard reference). We will henceforth 
denote by dJ{uo) the subdifferential of J at mq € i-e., the set of all ^ G X* such 
that 

J(u) > J{uq) + (^, w — uq) _Y , for all u G X. 

In this case, we call ^ a subgradient of J at uq- We denote by K*: H X* the adjoint 
operator of K , where we identify the Hilbert space H with its dual H* by means of 
Riesz' representation theorem. Under Assumption 12 . II it is guaranteed that solutions 
of ((ll exist for all g S K{D{J)) and that the iteration ([3a|) is well defined. The proof 
is analogous to (Frick & Scherzer 2010, Lem. 3.1). 

Recall that the Legendre-Fenchel conjugate J*:X* M of J is defined by 
J*(x*) = sup^gjsf {x*,x)-^t X ~ Jix). The dual problem to ([ij is then defined by 



inf 

peH 



r{K*p)-{p,g) . (6) 



Sufficient and necessary conditions for guaranteeing the existence of a solution u'^ E X 
of (H)) and a solution p^ E H oi ^ are the Karush-Kuhn-Tucker conditions, which 
read as 

K*p^ e dJ{u'^) and Ku"^ = g. (7) 
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From an inverse problems perspective, these conditions are understood as source 
conditions (Burger & Osher 2004) that dehmit a class of particular regular solutions 
of ((!]) that can be reconstructed from noisy data at a certain rate depending on 
the noise level 6. If the source condition ([7]) does not hold, then solutions of ([1]) may 
still exist (e.g. if Assumption 12.11 holds) whereas ^ has no solutions. The value of 
([6|), though, will still be finite: 

Lemma 2.2. Assume that Assumption 12.11 holds and let g X be a solution of ([1]). 
Then 



inf 



r{K*p)-{p,g) ^-J{u 



Proof. Define a function FtX x — > K by setting T{u,p) = J(u) if Ku = g + p 
and G{u,p) = +oo else. According to (Ekeland & Temam 1976, Chap III. Prop. 
2.1) the assertion holds, if the function p i-7> h{p) = miuex^{u,p) is finite and 
lower semicontinuous at p = 0. Since p{0) — J(ut) < cxd it remains to prove 
lower semicontinuity. Let therefore {pk}].^f^ be a sequence in H such that p^. — )■ 0. 
Without loss of generality, we may, after possibly passing to a subsequence, assume 
that h{pk) < oo for every fc, which amounts to saying that the equation Ku = g + Pk 
has a solution Uk € X satisfying J{uk) < oo. In addition, because of Assumption 12. 1[ 
we can choose Uk such that the infimum in the definition of h is realized at u^, that 
is, h{pk) = T{uk,Pk)- 

Now, if J{uk) — > oo as fc — >■ oo, nothing remains to be proven. Thus we can 
assume that there exists a subsequence of {ufc'} such that supf,,^jq J{uk') < oo. It 
is not restrictive to assume that lim.k'^00 J (uk') = liminffe_j.oo >/(u/c)- Moreover, we 
observe that \\Kuk — g\\'^ = \\pk\\'^ is bounded, since pk 0. Thus it follows from 
Assumption 12. ll that there exists a further subsequence {uk"} such that Uk" — ^ u for 
some u € X. This implies that Kuk" Ku = g, and the lower semicontinuity and 
convexity of J finally proves that 



liminf /i(pfc) — lim J(uki) = liminf J(uk") > J{u) > >^(^^) — h{0). 

A:— >oo k'^QO k"^oo 



□ 



Similar to the duality relation between the optimization problems ([T]) and 
(O such a relation can be established for the ALM: As it was first observed in 
(Rockafellar 1974), the dual sequence {pQ,pf,...} generated by the ALM can be 
characterized by the proximal point method (PPM). To be more precise, for all fc > 1, 



Pk = argmin 



l^\\p~pi-i\\ +n{r{K*p)^{p,g')) 



(8) 



The PPM was introduced by Martinet in (Martinet 1970) for minimizing a convex 
functional, which in the present situation is the dual functional (jS]). The sequence 
{Pk} generated by the PPM is known to converge weakly to a solution of ^ if it 
exists, i.e., when ([7]) holds. If this is not the case, then stiU J*{K*pl) - {pi,g^) 
converges to the value of the program (jB)) which, in the general case, may be —00, of 
course. 
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3. Convergence Rates 

It is well known that linear convergence rates (with respect to the Bregman distance) 
for iterates of the ALM can be proven if the source condition ^ holds (cf. (Burger 
et al. 2007, Frick & Scherzer 2010)). In this section we prove lower order rates of 
convergence in the case, when the source condition (O does not hold. Instead, we 
impose weaker regularity conditions on solutions of ([Ij in terms of variational 
inequalities. We formulate this in the following 

Assumption 3.1. We are given an index function $; [0, oo) — > [0,oo), i.e., a non- 
negative continuous function that is strictly increasing and concave with $(0) — 0. 
Moreover, D:X x X — > [0, cx)] satisfies D[u,u) = whenever u £ X, and is a 
solution of (QP is such that 

£>(?/, ut) < J(it)-J(ut) + $(||A'M-g||2) for all u ex. (9) 

We denote by 5* the Legendre-Fenchel conjugate of . 

A typical choice is D{u, v) — l3D^j{u, v), where (3 E (0, 1] and 

D${v, u) = J{v) - J{u) ~{(,v~ u)^.,, (10) 

is the Bregman- distance of u and v w.r.t. ^ G dJ{v). With this, Q is equivalent to 
the condition 

{^\u^ - u)^,^^ < (1 - /3)i?«; (u, yt) + ^{\\Ku - gf) (11) 

for all u € In this form, variational inequalities have been introduced in (Hofmann 
et al. 2007, Scherzer et al. 2009) with $(s) — -^s, and for general index functions 
in (Bot & Hofmann 2010, Grasmair 2010). 

The following theorem asserts that the condition (|9]) in Assumption 13.11 imposes 
sufRcient smoothness on the true solution that the iterates of the ALM approach 
with a certain rate (that depends on $). 

Theorem 3.2. Let Assumvtions \2. 1\ and \3. 1\ hold. Then, there exists a constant C > 
such that 



and 



D{ulu^)<Ct^ 0+^') 



\Kui-g'f<cU'^'' 



^71 



In particular, if tn ^ W^rjjrj, then 



D{ulu^)^0[-J^A and llif < - = 0(<5' 



^-1(^2)^ 

Theorem 13.21 is a consequence of the following two Lemmas. 
Lemma 3.3. Let Assumptions 12.11 and 13.11 hold and define for p E H , t > Q and 5 >Q 

^ip,t,S)^ (^t^(16/t) + tS' + r{K*p) + Jiu^) - {p,g) + ^^^ . 

Then, there exists a constant C > such that 

Diulu^)<Cij{p,tr.,S) and \\Kui^ g'\\^ < C^^iE^^ (12) 

for all p ^ H. 
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Proof. Without loss of generality we assume that — and we shall agree upon 
G{p,g) = J*{K*p) - {p,g). In (Giiler 1991, Lem. 2.1) it was proved that for aU p e ^ 

||Pn ~ ^ ^, S\ r^i 8 &\ I Ibll /iQ\ 

— - — 7r-2 — < G(P:g ) - G(p„,g ) - " + (1^) 

Since G(p,5*) - G(p^,g^) = G(p,5) - G{p>tg) + (p - g - g*) and -p^i = 
''"(ff'' ~ this implies that 

I llifuf, - /If < Gb,g) - G{plg) - + + _ _ 

< G(p, g) + J(^t) „ ll^' P-ll + + _ p^, 5 _ /) , (14) 



where the second inequahty fohows from Lemma l2.2[ Setting p — p^^ this proves that 

t 2 11-^11' 

f \\Kui -g'W < r{K*pi) - (pig) + J{u^) 



Since K*pf^ e dJ{ufJ, we observe that J*{K*pl) + J{ui) = wf,) and conclude 

that 

I ||ifuf, - < J(«t) _ ) + , Kui - .g) + ML 

- j(7.t) - j(/j + (pI Kui - g') + {vl g' - g) + 



2/tri 

1 



Applying Young's inequality (a, 6) < ||a|| /2 + ||6|| /2 first with a ~ ^Ijtnpi and 
h = {Kui ^ g^)\/tn/2, and then with a = pij \ft^ and h = -JUi{g^ — g), wc obtain 

^ \\Kui -gf < J(«t) - Jiui) + {pig' -g) + ^ 



4 II „ . II _ V . V . \.n,. 2tn 

X2f 2lln'^ll^ 
< J(mT) - J(u„) + — - + ■ " 



2 ^TT, 

Summarizing, we find that 

Now, we observe from © that 7(1*^) - J(w^) < -L)(w^,Mt) + ^\\Kui~g\\ ). 
Plugging this inequality into the above estimate yields 

11^^^ - + r^(^"' ^ r'^dl^"' - sf) + + (15) 

Since \1/ is the Legendre-Fenchel conjugate of < H> ^~^{t), i.e., ^'(s) = supj>Q — 
$~^(t), it follows that st < vl'(s)+$"^(i) for all s,t>0, and in particular, for t = $(r). 
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that s<i>(r) < ^'(s) + r for all s,r > 0. Setting s ~ 16/t„ and r = \\Ku^^ — g^\\ gives 



r^i\\KK 



1 16 ,11 A l|2, 

--<i>(||X<-5|| ) 









< 
















^16 


< 













\Ku^-g\ 



\Kui~g' 



Combining this with (|15p yields 



55^ 8^ 
2 £ 



(16) 



Finally, we observe again from (IT3l) that for all p e iJ 



lb-Pn|| ^ G{p,g')-G{plg') ^ \\p\ 



2tl 



< 



< 



t ' 2f2 

G{p,g)-G{ptg) ^ 1 



IIpII 



in 



+ — {p-Pn,9-9 ) + ^ 



G{p,g) -MgevG{q,g) \\p ~ pf, 



4.tl 



2tl 



This shows that 



Wn 



< 



P-Pn 



\\P\\ 



^ G(p,g ) - infgey G{q,g) 



3Mf 



Combining p7p with (IT51) and applying Lemma 12.21 finally gives 



(17) 
(18) 



2 II " ^ II <„ ^ ^ - 4 I i„. 



1 ^ /16 
< — 

" 4 V tr,. 



t^ 



^^ G{p,g)-migevG{q,g) ^ 133(5^ , 48 ||p||^ 



t^ 



□ 



Lemma 3.4. Let Assumptions 12.11 and 13.11 hold. Then, 



inf 

pen 



r{K*p) + J{u^)-{p,g) + 



2t 



Proof. Classical duality theory (see (Ekeland & Temam 1976, Chap III)) implies that 



u := inf 



J*{K*p) + J{u^)-{p,g) + 



\\P\\ 



2t 



= — inf 



^-\\Ku-g\f + Jiu)-Jiu^) 
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as the right hand side of this equation is the dual of the left hand side. Using the 
variational inequality ^ and the non- negativity of _D, we therefore find that 



H < sup 
< sup 



m\Ku-g\\')-D{u,u^)--\\Ku-gr 



'f(\\Ku-g\ 



*^\\Ku-gf 



Replacing ||A'u — gH" by s > in the last term and using the definition of we 
obtain 



H < sup 

s>0 



t 

TTSUP 

^ s>0 



2$(s) 



t 

= sup 

^ s>0 



which proves the assertion. 



□ 



We close this section by a statement concerning the dual variables {pi,P2: • ■ •} 
generated by the ALM. It is well known (in the case when (5 = 0) that these stay 
bounded if and only if the source condition ^ holds. Assumption 13.11 however, 
allows to control their growth, as the following result shows. 

Corollary 3.5. Let Assumptions 12.11 and 13.11 hold. Then, there exists a constant 
C > such that 

iKir<«j 

Proof. It follows from ([T7| that there exists a constant C > such that 
\\pi\\' < Ct„ (^r{K*p) + Jiu^) - {p,g) + M! +<„^2^ 
for all p € H . Applying Lemma [3.41 vields the desired estimate. □ 



4. Morozov's Discrepancy Principle 

In this section we study Morozov's discrepancy principle as an a posteriori stopping 
rule for the ALM. To be more precise, if {■u^,U2,...} is generated by the ALM, 
Morozov's rule suggests to stop the iteration at the index 

n*(,5) = min{n e N : \\Kui - g^\\ < p5] , (19) 

where p > 1. In this section we prove convergence rates for the iterates u^.j^^-j given 
that Assumption 13 . 1 1 holds . Morozov's principle for the case when the source condition 
([7]) holds was studied in (Frick et al. 2011). Theorem 14.21 below extends this result to 
regularity classes that are delimited by the variational inequality in Assumption 13 . 1 1 
Additionally to these, we will assume 

Assumption 4.1. Let Assumvtion \3.1\ hold. 

(i) The mapping s $(s)^/s is non-increasing. 

(a) The sequence of stepsizes {ri, r2, . . .} in the ALM is bounded. 
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Theorem 4.2. Let As sumptions and \4.1\ hold and assume that n*{5) is chosen 
according to Morozov's discrepancy principle M9\) for some p > 1. Then there exists 
a constant C > independent of p such that 

Remark 4.3. Assume that the variational inequahty © is satisfied with $(s) — Cs^ 
for some C > and p > 0. Then, setting u = u'' + tz for some z ^ X and t > 0, the 
non-negativity of D imphes in particular the inequality 

J(ut) - J(ut + tz) < Ct^P \\KzfP . 

Now assume that p > 1/2. Then we obtain, after dividing by t and considering the 
limit t — > 0+, that the directional derivative of J satisfies —J'{u^){z) < 0. Because 
z was arbitrary, this implies that minimizes the regularization term J. Thus the 
variational inequality can hold in non-trivial situations, if and only if p < 1/2. 

Now note that the same condition is required for the function $(s)^/s = C^s^p~^ 
to be non-increasing. Therefore, in the case of a variational inequality of Holder type, 
Assumption 14.11 imposes no relevant further restrictions on the index function. 

Before we give the proof of Theorem l4.21 we state the following Lemma, which is 
interesting in its own right. 



Lemma 4.4. Let Assumptions 12.1 1 and 13711 hold and assume that n*{S) is chosen 
according to Morozov's discrepancy principle (|19p. Then, 

2 

Proof. Without loss of generality we may assume that n*{d) > 1; else the assertion is 
trivial. Denote for the sake of simplicity n := n*{6) — 1. Then it follows from ([TT 

that ^5*^11^ > P^^^ ■ Plugging in this relation into (|14|) yields 



2 ^t'f-[ 



2 



< r{K*p) - (p, g) + Jiu^) + ^ + (p _ 3 _ /) 



for every p ^ H. Applying Young's inequality 



(p-plg-g') 



< 



\P-Pn\\ , tnS'^ 



2tf. 

we obtain with Lemma [57^ the estimate 



< inf 

peH 



r{K*p)~{p,9) + J{u^) + 



" 2 \tn 



2^n 

This proves that {p^ — 1)6^ < 5'(2/t,-j). Now the assertion follows by applying the 
monotoneously increasing function to both sides of this inequality and adding 
the last step size r„.(5). □ 



Next we need another lemma, which relates the condition on $ in Assumption 1471 
to an equivalent condition on the function = (<i>~^)*. 
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Lemma 4.5. Let $ be an index function and ^ the Fenchel conjugate of Then 
the mapping s i— ?► <i>(s)^/s is non-increasing, if and only if the mapping t i-^ t'^'i'{2/t) 
is non-decreasing. 

Proof. First note that, by means of the change of variables t ^ 2/t and ignoring 
the constant factor, the mapping t i~> t^'^{2/t) is non-decreasing, if and only if the 
mapping 1 1-> H{t) '■— '^{t)/t^ is non-increasing. Because 5* is convex and continuous, 
this condition is satisfied, if and only if H'{t) < for every t > for which 
exists. Now, 

and therefore H'(t) < if and only if - 2*(t) < 0. Now recah that ^ is the 

Fenchel conjugate of and therefore = + <^-^(^'{t)). Thus H'(t) < 0, 

if and only if - < 0. 

Similarly, the mapping s i— > $(s)^/s is non-increasing, if and only if the mapping 
s !—>■ H{s) :— s^/$~^(s) is non-increasing, which in turn is equivalent to the condition 

M'( ) = 2,s _ s^<^>-''{s) ^ s{2^-\s)-s^-^'{s)) 

Because of the equality s$^^'(s) = "I>^^(s) -I- 5'(<i>~^'(s)) , this is the case, if and only 
if $^^(s) - vl/($-i'(s)) < 0. The assertion now follows from the fact that s = "^'{t) 
if and only if < = "I>^^'(s), which, again, is a consequence of the fact that and 
are conjugate. □ 

Proof of Theorem \4-2\ Throughout the proof we use the abbreviation n = n*{5). First 
observe that K*pl~e dJ{ui) and thus J(u^) - J(u'l') < {pi,Kui - g). From the 
discrepancy rule (fT9| it follows that 

\\Kui-g\\ < \\Kui-g'\\+d<ip + l)S, 

and hence the variational inequality © implies 

D{ulu^)<\\pi\\{p+l)5 + <P{{p+irS'). (20) 



As in the proof of Lemma [3731 we observe that for all s, r > one has s$(r) < ^'(s) + r. 
Setting r — {p + 1)'^6^ and s = ^^^((p^ — 1)6'^), one finds, after dividing both sides of 
the inequality by s, that 

which yields an estimate for the second term in (|20p. For estimating the first term, 
we note that CoroUarv 13.51 implies the estimate 

\\pi\\<CtnU(f) +SA ^ <Ctn^(p\ ^ +Ct,J 
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for some constant C > 0. By assumption, the mapping x i— $(x)^/a; is non-increasing, 
and therefore, using Lemma [4.51 the mapping s n- s'^'i'{2/s) is non-decreasing. Thus 
we obtain, after using the estimate for t„ of Lemma [4.41 and the monotonicity of ^, 



^"11 *'H(p^-l)'52)^^V'^V2 + r„*-i((p2_l)52) 

2CS 



1'-l((p2 - 1)52) 



+ CTnS 



Consequently we have 

)< ^_l^(^2_l)52) +^-l((^2_l)52) 
^ 2(C-fl)(p-f 1)2^2 ^^2.2 

which proves the assertion with C:=2(C + 1). □ 



5. Examples 

In this section we discuss particular instances of the variational inequality (jO]) and 
the implications of the general results in Sections [3] and |4] for these special scenarios. 
The first two examples shed some light on the relation of variational inequalities 
and more standard notions of source conditions: the KKT condition ([7]) and Holder- 
type conditions. The third example shows an example from sparsity promoting 
regularization, where standard notions of source conditions together with an additional 
restricted injectivity assumption allow the derivation of convergence rates with respect 
to norm instead of the Bregman distance. 



5.1. Standard Source Condition 

It is quite easy to see that the standard source condition ([7]) implies the variational 
inequality (fTTj) . Indeed, assume that is a solution of ^ and that K*p^ e dJ{u^) 
for some £ H. By defining — K*p^ one observes 

{^\u^ -u)^,^^^{p\g-Ku) < \\p-^\\\\Ku-g\\. 

Setting /3 = 1 and $(i) = ||pt|| ti/2 gj^es pT|) . 

The converse is in general not true, i.e., ([TT|) with $(i) — jt^^^ (7 > 0) does not 
imply the existence of a ^ V such that K*p'' S dJ{u^). However, if (|lip is replaced 
by the stronger condition 

{e,u^ - u)^,^^ < (1 - l3)Dj{u, u^) +^\\Ku- g\\ , (21) 

for all u X , then the two notions are equivalent. Here, Dj{u,v) = J{u) — J{v) — 
J'{v){u — v) and J'{v){w) is the directional derivative of J at w in direction w: 

J'(t;)H = lim ^{J{v + hw)- J{v)). 
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Note that for convex J, the directional derivative is weh-defined for every v and 
w (though it takes values in [— cx),cx)]) and is positively one-homogeneous, i.e. 
J'{v){tw) = tJ'{y){w) for all t > 0. 

In order to see the aforementioned equivalence, let w G X and set u = — in 
(|2T|l for some t > 0. Then, 

(e^ ^ (1 - - + 7 \\tKv\\ . 

Since the mapping w i— > J'{u'^){w) is positively one- homogeneous, this implies that 

(en < (1 - /3) „ j'(.t)(_„)) + , ,|,,,,,| , 

for all f G X and t > 0. Letting i 0+ this shows that (C^w)^. x — ^ H-^^H 

V G X and hence iir*pt — for some € H according to (Scherzer et al. 2009, Lem. 

8.21). 

In the particular case where the mapping J is Gateaux differentiable at u^, the 
subdifferential dJ{u^) contains a single element ^ which coincides with the directional 
derivative, that is, (^^w) = J'{u^){v) for every v ^ X. Thus, in this case, the source 
condition is equivalent with the variational inequality. 

If ^[t) = 7t^/^ then the Fenchel conjugate ^I^ of reads as '^{t) = -^/{2^/2)t^. 
Hence it follows from Theorem 13 . 2 1 that there exists a constant C > such that 

given the a priori stopping rule t„ x . This is the well known convergence 
rate result for the standard source condition (see (Burger et al. 2007, Frick & 
Scherzer 2010)). We note that the results in (Frick & Scherzer 2010) are slightly 
stronger, as they give (5-rates for the symmetric Bregman distance (see also (Frick 
et al. 2011)). If Morozov's discrepancy principle (|19l) is applied as an a posteriori 
stopping rule, we obtain from Theorem 14.21 that 



Df"' (4.(5) , u^) < CJ ^-^-^S + C{p + If 6^ sup Tfc. 

y P — t feeN 

This coincides with the results in (Frick et al. 2011, Thm. 4.3), where Morozov's 
discrepancy rule for the standard source condition was studied. 



5.2. Holder-type Conditions 



In this section we study the relationship between the variational inequality (jlip and 
Holder- type source conditions for the iteration (|4]). 

We first consider the case of the iterated Tikhonov method, i.e., i = Id and thus 
J{u) = i Then, a solution ut of ([1]) is said to satisfy a Holder condition with 

exponent < < ^ if (K*KYp'^ = = dJ(v)). If satisfies a Holder condition 



with exponent then pT|) holds with I?j (u,w^^) — 5 — and $(3) x 
To see this, observe that the interpolation inequality (cf. (Engl et al. 1996, 
implies 



S 1+2" . 

p.47)) 



< 



< 



.\P 

¥ 

2^ 



2i/ 



[K* K)^ {v!^ - u) 
p'^\\{\\Ku-g\;yDi{u,u^y 



l-2f 
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Using Young's inequality ab < dP jp + h'^ jq with q — 2/(1 — 2v) and p = 2/(1 + 2i^) 
shows for all > 

{||A-„-<,ir)''i33'(«,«')'=^ = \mu-,jf)\Di{.v.,u^)^ 

2,^(1+2,.) z 

Choosing ?7 such that 1 — /3 = t^t^^ (i^^)2^T^ < 1 results in (fTT|) after setting 
$(s) = csTTfe with c = ^^.^y+^r..) 2^ • 

In case of the iterated Tikhonov-Morozov method, we consider ([4]) with Id 
and L: D{L) C X ^ H being a densely defined, closed linear operator. Recall that in 
this case L = (Id + LL*)^^ and L = (Id + L*L)~^ are self-adjoint and bounded linear 
operators (cf. (Groetsch 2007, Chap. 2.4)). A solution of (P) is said to satisfy 
a Holder condition with exponent < < 5 if Lu^ = L'^oj^ for some a;^ G H. We 

show that this condition implies ^ when D{u,u^) equals ^ ||-^^^~ -^""^11^ (^'-"" some 
7 € (0,1)) whenever m e D{L) and +00 else. To see this, recall that J{u) = 00 if 
u ^ D{L). Thus (O is equivalent to 

{Lu\Lv) - Lu) < (l-7)||Lw-iut||2_,_$(||y_yt||2) (22) 

for all u £ D{L). Setting Lu^ = L'^uj'^ shows together with the interpolation inequality 
and (Groetsch 2007, Lem. 2.10) that for all u € D{L) 



{Lu\Lv) - Lu) = (uj^L^iLv) - Lu)^ 



L^{Lv) - Lu) 



1v 



\Lu^ — Lu 



I \-1v 



/II til II r f i||2i^ II t ll^"" II r t T l|l"2i/ 

< ||a;' II ||LL2 II ||u'— m|| ||Lu'— Lu|| 

With the same arguments as in the case of the iterated Tikhonov method above, we 
conclude that holds with $(s) = cs^^'+i for some constant c > 0. 

Now let again be X a general Banach space and J: X M be convex such that 
Assumptions 12.11 are satisfied. As revealed by the calculations above, the variational 
inequality (|9]) with $(s) x s ^+^" can be seen as a generalized Holder condition. Note, 
that in this case the Legendre conjugate ^ of comes as 5'(i) x i^^^" and thus 
Theorem 13.21 amounts to say that there exists a constant C > such that 

D{ui,u^) < C6^ 

-2 

if tn X S ^+^" and Morozov's discrepancy principle p9)) shows that 

ut) < C \ + C{p + If 5^ supTfe. 

These results coincide with the lower order rates for the iterated Tikhonov method 
(Hanke & Groetsch 1998) and iterated Tikhonov-Morozov method (Groetsch 2007). 



5.3. Sparsity Promoting Regularization 

We now discuss the application of the results derived in this paper to sparsity 
promoting regularization. To that end, we assume that AT is a Hilbert space with 
orthonormal basis {(pi : i G N}, and we consider the regularization term J{u) :— 
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J2i\{'t'i:U)\'^ for some 1 < q < 2 (see (Daubechies et al. 2004)). In (Grasmair 
et al. 2008), it has been shown that, for Tikhonov regularization, this setting allows the 
derivation of convergence rates of order 0(5^) with respect to the norm, if satisfies 
the standard source condition K*p^ e dJ{u^) for some e -ff, and, additionally, a 
restricted injectivity condition holds. In the following, we will generalize these results 
to the Augmented Lagrangian Method and source conditions of Holder type. 

Assume that there exists < < 1/2 such that {K*KYp^ = G dJ[u^) and 
that supp(u^) := {i G N : {4>i,u) ^ 0} is finite. In case q > 1 assume in addition 
that the restriction of K to span{(/)i : i S supp(x^)}, and in case q = 1 assume that 
the restriction of K to span{(/)i : |(</)i,^'l')| < 1} is injective. We will show in the 
following that, under these assumptions, there exists a constant C > such that ([9]) 
holds with D{u,u^) = C — and $(s) x sF^fe^ in case q > 1, and with 
D{u, u^) = C '^u^ — u|| and ^(s) x for q = 1. 

It has been shown in (Grasmair et al. 2008, Proofs of Thms. 13, 15) that the 
given assumptions imply the existence of constants Ci , C2 > such that 

Ci - uf < C2 \\Ku - gll" + J{u) - J{u'^) - {S},u - u^) 

for all u € X. Applying the interpolation inequality to {i^\u — u^^, we obtain, 
similarly as in Section W% the estimate 

Ci - uf < C2 \\Ku - + J{u) - J{v)) + IIp^II \\Ku - gf" - uW^'^" . 
Now Young's inequality with p — q/{l — 2v) and p,, = q/ {q — 1 + 2v) shows that 

Ci - ^—^rjT^] llut - uf < C2 WKu-gW'' + J{u) - J{u'^) 



q 



I III (? - 1 + 2j/ 2 II I 

\\P \\ 779-1+2- \\Ku-g\ 



Choosing 77 > such that C — Ci — \\p^\\ ^—;f^V^'^" > and setting 

*/^ ^ i Iltll9^1 + 2J^ 1 2,.g 

$(s) = C2S2 + 7^,-1+2,. s, -1+2,. ^ 

q 

we obtain the variational inequality ([9]). Because q3il2v — 9' asymptotic 
behaviour of $ for s — is governed by its second term, which shows that 
$(s) X s 9-1+21- , Moreover, in the special case q = 1, the term s g-1+2- reduces to 
s^ independent of the type of the source condition. For the function ^, we obtain 
the asymptotic behaviour ^(s) x s 9-i+(2-g),. . Thus, Theorem 13.21 shows that for 
tn ^ q-i+2u have the estimate 

for (5 > sufficiently small, and a similar estimate for Morozov's discrepancy principle. 

Remark 5.1. In (Grasmair et al. 2011), it has been shown for Tikhonov regularization 
with J{u) — \ {(j)i,u)\, which is the special case of the ALM with a single iteration 
step, that a linear convergence rate with respect to the norm is equivalent to the usual 
source condition. Thus the results above imply that, in the case q = 1, the Holder 
type source condition {K*KYp^ e dJ{u^) in fact already implies the standard source 
condition K*p^ G dJ{u^) for some different source element p^ . 
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